Skip to content

[Feature] kl_mask_threshold #3208

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 7 commits into
base: gh/vmoens/163/base
Choose a base branch
from
Open

[Feature] kl_mask_threshold #3208

wants to merge 7 commits into from
+152 −1

Conversation

@vmoens
Copy link
Collaborator

@vmoens vmoens commented Oct 16, 2025 •
edited
Loading

@vmoens
Update
9379070 
[ghstack-poisoned]
@pytorch-bot
Copy link

pytorch-bot bot commented Oct 16, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/3208

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

❌ 5 New Failures, 1 Cancelled Job, 16 Unrelated Failures

As of commit 94b5fc0 with merge base 13434eb (image):

NEW FAILURES - The following jobs have failed:

CANCELLED JOB - The following job was cancelled. Please retry:

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Oct 16, 2025
This was referenced Oct 16, 2025
Merged
Merged
Open
Open
@vmoens
Update
73a3c5b 
[ghstack-poisoned]
@vmoens vmoens mentioned this pull request Oct 23, 2025
Merged
@vmoens
Update
edaf07b 
[ghstack-poisoned]
@vmoens vmoens mentioned this pull request Oct 23, 2025
Open
@github-actions
Copy link

github-actions bot commented Oct 23, 2025 •
edited
Loading

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 154. Improved: $\large\color{#35bf28}26$. Worsened: $\large\color{#d91a1a}1$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_tensor_to_bytestream_speed[pickle] 83.1656μs 81.4282μs 12.2808 KOps/s 12.0519 KOps/s $\color{#35bf28}+1.90\%$
test_tensor_to_bytestream_speed[torch.save] 0.1409ms 0.1401ms 7.1373 KOps/s 7.0091 KOps/s $\color{#35bf28}+1.83\%$
test_tensor_to_bytestream_speed[untyped_storage] 0.1219s 0.1217s 8.2161 Ops/s 7.3530 Ops/s $\textbf{\color{#35bf28}+11.74\%}$
test_tensor_to_bytestream_speed[numpy] 2.9264μs 2.9156μs 342.9856 KOps/s 339.2323 KOps/s $\color{#35bf28}+1.11\%$
test_tensor_to_bytestream_speed[safetensors] 42.1903μs 42.0263μs 23.7946 KOps/s 23.0332 KOps/s $\color{#35bf28}+3.31\%$
test_simple 0.5521s 0.5495s 1.8198 Ops/s 1.6917 Ops/s $\textbf{\color{#35bf28}+7.57\%}$
test_transformed 1.1119s 1.1104s 0.9006 Ops/s 0.8703 Ops/s $\color{#35bf28}+3.48\%$
test_serial 1.6676s 1.6637s 0.6011 Ops/s 0.5633 Ops/s $\textbf{\color{#35bf28}+6.70\%}$
test_parallel 1.1618s 1.1070s 0.9034 Ops/s 0.9009 Ops/s $\color{#35bf28}+0.28\%$
test_step_mdp_speed[True-True-True-True-True] 0.2221ms 43.8210μs 22.8201 KOps/s 22.3112 KOps/s $\color{#35bf28}+2.28\%$
test_step_mdp_speed[True-True-True-True-False] 0.4287ms 25.1991μs 39.6840 KOps/s 40.1379 KOps/s $\color{#d91a1a}-1.13\%$
test_step_mdp_speed[True-True-True-False-True] 0.4346ms 25.1074μs 39.8289 KOps/s 39.1205 KOps/s $\color{#35bf28}+1.81\%$
test_step_mdp_speed[True-True-True-False-False] 51.2510μs 13.9360μs 71.7565 KOps/s 71.6374 KOps/s $\color{#35bf28}+0.17\%$
test_step_mdp_speed[True-True-False-True-True] 0.4593ms 48.0528μs 20.8105 KOps/s 20.9141 KOps/s $\color{#d91a1a}-0.50\%$
test_step_mdp_speed[True-True-False-True-False] 0.4438ms 27.8858μs 35.8606 KOps/s 34.9577 KOps/s $\color{#35bf28}+2.58\%$
test_step_mdp_speed[True-True-False-False-True] 0.4588ms 28.0344μs 35.6705 KOps/s 35.6208 KOps/s $\color{#35bf28}+0.14\%$
test_step_mdp_speed[True-True-False-False-False] 53.9910μs 16.7979μs 59.5312 KOps/s 58.8277 KOps/s $\color{#35bf28}+1.20\%$
test_step_mdp_speed[True-False-True-True-True] 0.4642ms 51.0022μs 19.6070 KOps/s 19.7245 KOps/s $\color{#d91a1a}-0.60\%$
test_step_mdp_speed[True-False-True-True-False] 0.4351ms 30.5372μs 32.7469 KOps/s 32.3541 KOps/s $\color{#35bf28}+1.21\%$
test_step_mdp_speed[True-False-True-False-True] 0.4362ms 27.8748μs 35.8748 KOps/s 36.0310 KOps/s $\color{#d91a1a}-0.43\%$
test_step_mdp_speed[True-False-True-False-False] 48.8600μs 16.6033μs 60.2290 KOps/s 59.2732 KOps/s $\color{#35bf28}+1.61\%$
test_step_mdp_speed[True-False-False-True-True] 0.4683ms 52.3395μs 19.1060 KOps/s 18.7140 KOps/s $\color{#35bf28}+2.09\%$
test_step_mdp_speed[True-False-False-True-False] 0.4431ms 32.7766μs 30.5095 KOps/s 30.0424 KOps/s $\color{#35bf28}+1.55\%$
test_step_mdp_speed[True-False-False-False-True] 58.9110μs 30.1187μs 33.2020 KOps/s 32.8391 KOps/s $\color{#35bf28}+1.10\%$
test_step_mdp_speed[True-False-False-False-False] 0.4238ms 19.3135μs 51.7771 KOps/s 50.8804 KOps/s $\color{#35bf28}+1.76\%$
test_step_mdp_speed[False-True-True-True-True] 86.8020μs 50.1424μs 19.9432 KOps/s 19.7734 KOps/s $\color{#35bf28}+0.86\%$
test_step_mdp_speed[False-True-True-True-False] 0.4321ms 30.5061μs 32.7803 KOps/s 32.5175 KOps/s $\color{#35bf28}+0.81\%$
test_step_mdp_speed[False-True-True-False-True] 2.4478ms 32.1697μs 31.0852 KOps/s 31.0704 KOps/s $\color{#35bf28}+0.05\%$
test_step_mdp_speed[False-True-True-False-False] 0.4336ms 18.3502μs 54.4954 KOps/s 54.1468 KOps/s $\color{#35bf28}+0.64\%$
test_step_mdp_speed[False-True-False-True-True] 0.4677ms 52.3198μs 19.1132 KOps/s 18.8762 KOps/s $\color{#35bf28}+1.26\%$
test_step_mdp_speed[False-True-False-True-False] 0.4395ms 32.9291μs 30.3683 KOps/s 29.7103 KOps/s $\color{#35bf28}+2.21\%$
test_step_mdp_speed[False-True-False-False-True] 69.3810μs 34.1657μs 29.2692 KOps/s 29.3399 KOps/s $\color{#d91a1a}-0.24\%$
test_step_mdp_speed[False-True-False-False-False] 0.4274ms 20.9915μs 47.6384 KOps/s 46.7413 KOps/s $\color{#35bf28}+1.92\%$
test_step_mdp_speed[False-False-True-True-True] 0.4709ms 55.0710μs 18.1584 KOps/s 17.7803 KOps/s $\color{#35bf28}+2.13\%$
test_step_mdp_speed[False-False-True-True-False] 0.4425ms 36.0347μs 27.7510 KOps/s 27.4240 KOps/s $\color{#35bf28}+1.19\%$
test_step_mdp_speed[False-False-True-False-True] 63.7610μs 33.9025μs 29.4964 KOps/s 29.3089 KOps/s $\color{#35bf28}+0.64\%$
test_step_mdp_speed[False-False-True-False-False] 0.4257ms 20.4261μs 48.9570 KOps/s 46.9210 KOps/s $\color{#35bf28}+4.34\%$
test_step_mdp_speed[False-False-False-True-True] 0.1124ms 57.7325μs 17.3213 KOps/s 17.0724 KOps/s $\color{#35bf28}+1.46\%$
test_step_mdp_speed[False-False-False-True-False] 66.6020μs 38.3608μs 26.0683 KOps/s 25.5187 KOps/s $\color{#35bf28}+2.15\%$
test_step_mdp_speed[False-False-False-False-True] 79.7720μs 36.5629μs 27.3501 KOps/s 27.2181 KOps/s $\color{#35bf28}+0.49\%$
test_step_mdp_speed[False-False-False-False-False] 55.0810μs 23.3772μs 42.7766 KOps/s 41.5790 KOps/s $\color{#35bf28}+2.88\%$
test_values[generalized_advantage_estimate-True-True] 11.1800ms 10.3148ms 96.9484 Ops/s 94.1136 Ops/s $\color{#35bf28}+3.01\%$
test_values[vec_generalized_advantage_estimate-True-True] 14.1815ms 11.3095ms 88.4215 Ops/s 88.6902 Ops/s $\color{#d91a1a}-0.30\%$
test_values[td0_return_estimate-False-False] 0.2538ms 0.1356ms 7.3758 KOps/s 7.6598 KOps/s $\color{#d91a1a}-3.71\%$
test_values[td1_return_estimate-False-False] 29.0951ms 28.1859ms 35.4788 Ops/s 34.5917 Ops/s $\color{#35bf28}+2.56\%$
test_values[vec_td1_return_estimate-False-False] 12.5573ms 11.4471ms 87.3586 Ops/s 88.0430 Ops/s $\color{#d91a1a}-0.78\%$
test_values[td_lambda_return_estimate-True-False] 43.8050ms 42.0274ms 23.7940 Ops/s 23.0851 Ops/s $\color{#35bf28}+3.07\%$
test_values[vec_td_lambda_return_estimate-True-False] 12.6036ms 11.3996ms 87.7223 Ops/s 87.7939 Ops/s $\color{#d91a1a}-0.08\%$
test_gae_speed[generalized_advantage_estimate-False-1-512] 9.3105ms 8.8697ms 112.7439 Ops/s 108.7601 Ops/s $\color{#35bf28}+3.66\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 1.8811ms 1.5365ms 650.8233 Ops/s 633.8013 Ops/s $\color{#35bf28}+2.69\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 0.4730ms 0.4195ms 2.3840 KOps/s 2.3696 KOps/s $\color{#35bf28}+0.60\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 28.3403ms 24.0493ms 41.5813 Ops/s 41.4605 Ops/s $\color{#35bf28}+0.29\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 2.0085ms 1.7423ms 573.9608 Ops/s 570.0848 Ops/s $\color{#35bf28}+0.68\%$
test_dqn_speed[False-None] 6.4632ms 1.4301ms 699.2496 Ops/s 685.3500 Ops/s $\color{#35bf28}+2.03\%$
test_dqn_speed[False-backward] 1.9983ms 1.9386ms 515.8234 Ops/s 514.6515 Ops/s $\color{#35bf28}+0.23\%$
test_dqn_speed[True-None] 0.6920ms 0.5293ms 1.8895 KOps/s 1.8943 KOps/s $\color{#d91a1a}-0.26\%$
test_dqn_speed[True-backward] 1.0200ms 0.9666ms 1.0346 KOps/s 993.4845 Ops/s $\color{#35bf28}+4.14\%$
test_dqn_speed[reduce-overhead-None] 0.6406ms 0.5136ms 1.9472 KOps/s 1.9335 KOps/s $\color{#35bf28}+0.71\%$
test_dqn_speed[reduce-overhead-backward] 0.9945ms 0.9541ms 1.0482 KOps/s 950.3619 Ops/s $\textbf{\color{#35bf28}+10.29\%}$
test_ddpg_speed[False-None] 3.2239ms 2.8782ms 347.4379 Ops/s 343.9728 Ops/s $\color{#35bf28}+1.01\%$
test_ddpg_speed[False-backward] 4.5782ms 4.1244ms 242.4585 Ops/s 239.1875 Ops/s $\color{#35bf28}+1.37\%$
test_ddpg_speed[True-None] 1.5153ms 1.3881ms 720.4175 Ops/s 697.3914 Ops/s $\color{#35bf28}+3.30\%$
test_ddpg_speed[True-backward] 2.4671ms 2.3795ms 420.2627 Ops/s 367.8807 Ops/s $\textbf{\color{#35bf28}+14.24\%}$
test_ddpg_speed[reduce-overhead-None] 1.6906ms 1.4003ms 714.1164 Ops/s 682.9888 Ops/s $\color{#35bf28}+4.56\%$
test_ddpg_speed[reduce-overhead-backward] 2.4639ms 2.3643ms 422.9601 Ops/s 363.0118 Ops/s $\textbf{\color{#35bf28}+16.51\%}$
test_sac_speed[False-None] 8.5254ms 8.0485ms 124.2473 Ops/s 125.6712 Ops/s $\color{#d91a1a}-1.13\%$
test_sac_speed[False-backward] 11.7956ms 11.2322ms 89.0295 Ops/s 88.8253 Ops/s $\color{#35bf28}+0.23\%$
test_sac_speed[True-None] 2.3104ms 2.0958ms 477.1377 Ops/s 455.4758 Ops/s $\color{#35bf28}+4.76\%$
test_sac_speed[True-backward] 4.0480ms 3.9543ms 252.8899 Ops/s 242.7950 Ops/s $\color{#35bf28}+4.16\%$
test_sac_speed[reduce-overhead-None] 2.3812ms 2.0954ms 477.2253 Ops/s 462.0444 Ops/s $\color{#35bf28}+3.29\%$
test_sac_speed[reduce-overhead-backward] 4.0723ms 3.9738ms 251.6499 Ops/s 215.9077 Ops/s $\textbf{\color{#35bf28}+16.55\%}$
test_redq_speed[False-None] 10.9317ms 10.3949ms 96.2011 Ops/s 95.9696 Ops/s $\color{#35bf28}+0.24\%$
test_redq_speed[False-backward] 18.8771ms 17.6990ms 56.5004 Ops/s 57.4854 Ops/s $\color{#d91a1a}-1.71\%$
test_redq_speed[True-None] 4.7253ms 4.3139ms 231.8068 Ops/s 228.3849 Ops/s $\color{#35bf28}+1.50\%$
test_redq_speed[True-backward] 10.8931ms 9.8006ms 102.0349 Ops/s 97.1072 Ops/s $\textbf{\color{#35bf28}+5.07\%}$
test_redq_speed[reduce-overhead-None] 4.6377ms 4.3327ms 230.8008 Ops/s 224.4098 Ops/s $\color{#35bf28}+2.85\%$
test_redq_speed[reduce-overhead-backward] 10.3276ms 10.0060ms 99.9400 Ops/s 100.3278 Ops/s $\color{#d91a1a}-0.39\%$
test_redq_deprec_speed[False-None] 11.5129ms 11.0753ms 90.2914 Ops/s 90.4769 Ops/s $\color{#d91a1a}-0.21\%$
test_redq_deprec_speed[False-backward] 17.4898ms 16.1830ms 61.7934 Ops/s 63.3116 Ops/s $\color{#d91a1a}-2.40\%$
test_redq_deprec_speed[True-None] 3.8242ms 3.6502ms 273.9585 Ops/s 271.8814 Ops/s $\color{#35bf28}+0.76\%$
test_redq_deprec_speed[True-backward] 7.7484ms 7.5202ms 132.9757 Ops/s 129.4181 Ops/s $\color{#35bf28}+2.75\%$
test_redq_deprec_speed[reduce-overhead-None] 3.9510ms 3.5799ms 279.3393 Ops/s 267.8512 Ops/s $\color{#35bf28}+4.29\%$
test_redq_deprec_speed[reduce-overhead-backward] 7.6590ms 7.4318ms 134.5571 Ops/s 129.5103 Ops/s $\color{#35bf28}+3.90\%$
test_td3_speed[False-None] 8.3115ms 7.9665ms 125.5259 Ops/s 118.2460 Ops/s $\textbf{\color{#35bf28}+6.16\%}$
test_td3_speed[False-backward] 11.3585ms 10.9146ms 91.6208 Ops/s 89.6811 Ops/s $\color{#35bf28}+2.16\%$
test_td3_speed[True-None] 1.8125ms 1.7795ms 561.9509 Ops/s 546.4656 Ops/s $\color{#35bf28}+2.83\%$
test_td3_speed[True-backward] 3.9246ms 3.5573ms 281.1126 Ops/s 224.6285 Ops/s $\textbf{\color{#35bf28}+25.15\%}$
test_td3_speed[reduce-overhead-None] 1.9829ms 1.7809ms 561.5202 Ops/s 557.5619 Ops/s $\color{#35bf28}+0.71\%$
test_td3_speed[reduce-overhead-backward] 3.6600ms 3.5629ms 280.6712 Ops/s 268.8945 Ops/s $\color{#35bf28}+4.38\%$
test_cql_speed[False-None] 28.7510ms 26.0390ms 38.4039 Ops/s 38.1614 Ops/s $\color{#35bf28}+0.64\%$
test_cql_speed[False-backward] 42.2189ms 35.4641ms 28.1975 Ops/s 28.2081 Ops/s $\color{#d91a1a}-0.04\%$
test_cql_speed[True-None] 12.7505ms 12.3368ms 81.0586 Ops/s 79.8905 Ops/s $\color{#35bf28}+1.46\%$
test_cql_speed[True-backward] 18.4300ms 17.9746ms 55.6341 Ops/s 53.3239 Ops/s $\color{#35bf28}+4.33\%$
test_cql_speed[reduce-overhead-None] 12.6965ms 12.3718ms 80.8289 Ops/s 81.0140 Ops/s $\color{#d91a1a}-0.23\%$
test_cql_speed[reduce-overhead-backward] 18.8125ms 18.3452ms 54.5102 Ops/s 54.8959 Ops/s $\color{#d91a1a}-0.70\%$
test_a2c_speed[False-None] 5.6910ms 5.4809ms 182.4533 Ops/s 180.8047 Ops/s $\color{#35bf28}+0.91\%$
test_a2c_speed[False-backward] 12.0913ms 11.8662ms 84.2727 Ops/s 83.2980 Ops/s $\color{#35bf28}+1.17\%$
test_a2c_speed[True-None] 4.7195ms 3.7529ms 266.4615 Ops/s 268.5513 Ops/s $\color{#d91a1a}-0.78\%$
test_a2c_speed[True-backward] 8.9960ms 8.6566ms 115.5194 Ops/s 114.2174 Ops/s $\color{#35bf28}+1.14\%$
test_a2c_speed[reduce-overhead-None] 4.0714ms 3.6796ms 271.7682 Ops/s 269.3588 Ops/s $\color{#35bf28}+0.89\%$
test_a2c_speed[reduce-overhead-backward] 8.9930ms 8.8012ms 113.6212 Ops/s 111.5288 Ops/s $\color{#35bf28}+1.88\%$
test_ppo_speed[False-None] 6.1667ms 5.7589ms 173.6433 Ops/s 167.3902 Ops/s $\color{#35bf28}+3.74\%$
test_ppo_speed[False-backward] 13.1037ms 12.5917ms 79.4173 Ops/s 78.7694 Ops/s $\color{#35bf28}+0.82\%$
test_ppo_speed[True-None] 4.0053ms 3.6353ms 275.0841 Ops/s 271.3358 Ops/s $\color{#35bf28}+1.38\%$
test_ppo_speed[True-backward] 8.7048ms 8.4414ms 118.4637 Ops/s 112.4235 Ops/s $\textbf{\color{#35bf28}+5.37\%}$
test_ppo_speed[reduce-overhead-None] 4.0516ms 3.6237ms 275.9619 Ops/s 272.2696 Ops/s $\color{#35bf28}+1.36\%$
test_ppo_speed[reduce-overhead-backward] 9.1014ms 8.6712ms 115.3244 Ops/s 110.5188 Ops/s $\color{#35bf28}+4.35\%$
test_reinforce_speed[False-None] 4.9563ms 4.5770ms 218.4851 Ops/s 219.3222 Ops/s $\color{#d91a1a}-0.38\%$
test_reinforce_speed[False-backward] 7.6168ms 7.3919ms 135.2830 Ops/s 134.9364 Ops/s $\color{#35bf28}+0.26\%$
test_reinforce_speed[True-None] 3.0528ms 2.8508ms 350.7790 Ops/s 333.6659 Ops/s $\textbf{\color{#35bf28}+5.13\%}$
test_reinforce_speed[True-backward] 8.1675ms 7.6942ms 129.9681 Ops/s 129.3495 Ops/s $\color{#35bf28}+0.48\%$
test_reinforce_speed[reduce-overhead-None] 3.2274ms 2.8298ms 353.3831 Ops/s 343.6148 Ops/s $\color{#35bf28}+2.84\%$
test_reinforce_speed[reduce-overhead-backward] 8.4109ms 7.9108ms 126.4088 Ops/s 119.5183 Ops/s $\textbf{\color{#35bf28}+5.77\%}$
test_iql_speed[False-None] 20.0695ms 19.5076ms 51.2621 Ops/s 47.8790 Ops/s $\textbf{\color{#35bf28}+7.07\%}$
test_iql_speed[False-backward] 31.1040ms 30.1889ms 33.1248 Ops/s 32.0760 Ops/s $\color{#35bf28}+3.27\%$
test_iql_speed[True-None] 8.5992ms 8.4010ms 119.0334 Ops/s 116.5144 Ops/s $\color{#35bf28}+2.16\%$
test_iql_speed[True-backward] 17.6958ms 16.5182ms 60.5394 Ops/s 59.0735 Ops/s $\color{#35bf28}+2.48\%$
test_iql_speed[reduce-overhead-None] 9.6201ms 8.4912ms 117.7694 Ops/s 112.8482 Ops/s $\color{#35bf28}+4.36\%$
test_iql_speed[reduce-overhead-backward] 17.5142ms 16.9230ms 59.0911 Ops/s 57.6489 Ops/s $\color{#35bf28}+2.50\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 7.4733ms 6.0279ms 165.8941 Ops/s 166.1161 Ops/s $\color{#d91a1a}-0.13\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.5420ms 0.2865ms 3.4910 KOps/s 2.6302 KOps/s $\textbf{\color{#35bf28}+32.73\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.6810ms 0.2785ms 3.5901 KOps/s 3.2218 KOps/s $\textbf{\color{#35bf28}+11.43\%}$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 5.9296ms 5.7271ms 174.6086 Ops/s 175.3938 Ops/s $\color{#d91a1a}-0.45\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 1.9408ms 0.2732ms 3.6601 KOps/s 2.8243 KOps/s $\textbf{\color{#35bf28}+29.59\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.5551ms 0.2527ms 3.9580 KOps/s 3.1793 KOps/s $\textbf{\color{#35bf28}+24.49\%}$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] 1.5226ms 1.2445ms 803.5186 Ops/s 718.8342 Ops/s $\textbf{\color{#35bf28}+11.78\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000] 1.5219ms 1.1627ms 860.0563 Ops/s 768.9479 Ops/s $\textbf{\color{#35bf28}+11.85\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 9.1911ms 6.0187ms 166.1493 Ops/s 171.0848 Ops/s $\color{#d91a1a}-2.88\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 2.1740ms 0.4259ms 2.3482 KOps/s 2.1076 KOps/s $\textbf{\color{#35bf28}+11.42\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.6301ms 0.4104ms 2.4366 KOps/s 2.2469 KOps/s $\textbf{\color{#35bf28}+8.44\%}$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 5.9683ms 5.7605ms 173.5973 Ops/s 175.5954 Ops/s $\color{#d91a1a}-1.14\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 1.9636ms 0.3344ms 2.9903 KOps/s 2.9727 KOps/s $\color{#35bf28}+0.59\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.4773ms 0.3218ms 3.1070 KOps/s 3.2354 KOps/s $\color{#d91a1a}-3.97\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 5.8585ms 5.6542ms 176.8592 Ops/s 174.1058 Ops/s $\color{#35bf28}+1.58\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.7591ms 0.3021ms 3.3097 KOps/s 3.0684 KOps/s $\textbf{\color{#35bf28}+7.86\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.6233ms 0.2844ms 3.5166 KOps/s 3.3195 KOps/s $\textbf{\color{#35bf28}+5.94\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 5.9823ms 5.8414ms 171.1907 Ops/s 169.0807 Ops/s $\color{#35bf28}+1.25\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 0.9100ms 0.4431ms 2.2569 KOps/s 2.2252 KOps/s $\color{#35bf28}+1.43\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.7417ms 0.4653ms 2.1493 KOps/s 2.3140 KOps/s $\textbf{\color{#d91a1a}-7.11\%}$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 6.4989ms 5.0485ms 198.0775 Ops/s 192.4471 Ops/s $\color{#35bf28}+2.93\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 8.8477ms 2.2667ms 441.1676 Ops/s 429.4938 Ops/s $\color{#35bf28}+2.72\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 7.2638ms 1.1701ms 854.6102 Ops/s 822.2521 Ops/s $\color{#35bf28}+3.94\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 0.4971s 14.9012ms 67.1089 Ops/s 54.6651 Ops/s $\textbf{\color{#35bf28}+22.76\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 11.1773ms 1.9145ms 522.3360 Ops/s 507.4021 Ops/s $\color{#35bf28}+2.94\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 6.8463ms 1.1567ms 864.5242 Ops/s 854.2490 Ops/s $\color{#35bf28}+1.20\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 6.7541ms 5.2297ms 191.2162 Ops/s 183.7740 Ops/s $\color{#35bf28}+4.05\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 10.4932ms 2.2157ms 451.3312 Ops/s 453.0982 Ops/s $\color{#d91a1a}-0.39\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 3.5734ms 1.2264ms 815.3907 Ops/s 751.4115 Ops/s $\textbf{\color{#35bf28}+8.51\%}$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-True] 35.5937ms 32.8218ms 30.4676 Ops/s 29.4261 Ops/s $\color{#35bf28}+3.54\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-False] 19.6758ms 17.8637ms 55.9795 Ops/s 55.5415 Ops/s $\color{#35bf28}+0.79\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-True] 35.9672ms 34.0462ms 29.3718 Ops/s 28.6101 Ops/s $\color{#35bf28}+2.66\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-False] 19.5528ms 17.7838ms 56.2311 Ops/s 54.4182 Ops/s $\color{#35bf28}+3.33\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-True] 37.6276ms 35.8363ms 27.9046 Ops/s 27.8859 Ops/s $\color{#35bf28}+0.07\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-False] 20.5674ms 19.0864ms 52.3934 Ops/s 50.6351 Ops/s $\color{#35bf28}+3.47\%$

vmoens added 3 commits October 23, 2025 11:45
@vmoens
Update
7dab9d1 
[ghstack-poisoned]
@vmoens
Update
15ff85b 
[ghstack-poisoned]
@vmoens
Update
1daa98f 
[ghstack-poisoned]
@vmoens
Update
94b5fc0 
[ghstack-poisoned]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. llm/feature

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@vmoens